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Fracture Critical POD Process Validation - 


Background 


¢ Flaw detection capability is established for inspection systems and inspectors on the basis 
of Probability of Detection (POD). The commonly accepted metric for an adequate 
inspection system is as follows: for a minimum flaw size which is smaller than the critical 
defect being sought, there is 0.90 probability of detection with 95% confidence (90/95 
POD). 


¢ A number of issues with current processes exist and will be addressed: 


1. The existing OSMA POD data set, including flaw standard specimens, have been consolidated 
at WSTF. Specimens are available upon demand. A variety of materials, geometries, defects, 
and NDE methods are included spanning about 860 specimens. 

Y Specimen storage procedures are inline with proposed best practices appendix.* 


2. In FY17 acentralized, searchable, and robust online database will be created to log and track 
POD specimens and results using agency mandated security and backup measures in line with 
ISO requirements. 


3. In FY18 a web-based service will be developed to enable analyses to be performed 
consistently across centers. 


* Reference: Proposed NASA-STD-5009 Appendix, TEST SPECIMENS AND THEIR IDENTIFICATION, HANDLING, CLEANING, ETCHING AND 
REFURBISHMENT. A 
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Anticipated Benefits 


To NASA funded missions: 


Although required and widely used to qualify inspection capabilities, the conduct of and 
tools for Probability of Detection (POD) analyses are not standardized for NASA and no 
system exists for cataloging POD data. 


¢ If we know a priori that a data set is inadequate for providing acceptable multi- 
parameter Maximum Likelihood Estimate (MLE) of POD, then what direction is need to 
be given and satisfied to make the data set a viable data set for multi-parameter MLE. 


e It’s better to know this before, rather than after the analyses is done and you are 
trying validate a POD estimate that cannot be validated. 
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Approach 


e Phase 0: Best practices appendices for POD and specimen storage (FY15) 
¢ First steps for the team’s education in POD. 


¢ Phase 1: Analyze the OSMA NDE POD Data Set (FY16) 


¢ Second step. Three studies completed, aided by Floyd Spencer. Reports in review. 


¢ Phase 2: Create a centralized repository for POD specimens/ NDE capabilities essentially 
providing a living, backed-up, and encrypted table for NASA-STD-5009 via MAPTIS (FY17) 


¢ Phase O and 1 efforts informed team of database field and organization requirements. 


e Phase 3: Develop and validate a web-based analysis tool standardizing analyses using the 
four common analysis algorithms: 

1. Multi-Parameter MLE 
a. Logistic Regression models (e.g. Logit and Probit) from MIL-HDBK-1823, NASA/TM—2014-218183 (April 2014). 
b. Add internal and external validation test for MLE estimates (draft coded into DOEPOD, but not released). 

2. Simple Binomial 
a. Method used by DOEPOD, and NASA Shuttle and Space Station Program estimate of POD. 
b. DOEPOD and DOEPOD’s validation was critically reviewed by statistician Bill Meeker/ lowa State.* 


C. NASA/TM-2014-218183 was critically reviewed, over a two year period by statisticians, Dr. William Q. Meeker of lowa State University and Dr. 
Floyd W. Spencer of Sfhire, and Dr. William Vesely for NASA. 


3. Bayes Rule 
4. Receiver/Relative Operating Characteristics (ROC) 


a. ROC is a display technology that may be based on Binomial, MLE, or Bayes rule, etc. 
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Accomplishments 


Team assisted in review of NASA-STD-5009, proposing appendices for POD Best 
Practices (Appendix B) and an appendix governing sample storage (Appendix C). 
Language requiring method validation added to -5009. 


Publication of DOEPOD methodology. Version 1.2 and manual available. The 
software was licensed under non-disclosure to organization directly supporting 
DOT and to a business venture for other applications. 


Phase 1: About 70% of the specimens have been tested. All non-Adobe Version 9 
files have been converted to MS Excel. Software glitch only allows Adobe 9 access 
to some inspector reports. Analysis and reports drafted for eddy current (ET), 
Level 3 (L3) Penetrant Testing (PT), and Level 4 (L4) PT. 


Phase 2: POD specimen database inclusion into MAPTIS via MSFC agreement 
began in FY17. Will poll Center POC’s for specimens and studies to import. 


¢ Worked in parallel: OSMA flaw specimens relocated to WSTF, being checked for cleanliness, 
cataloged, and secured for agency distribution. 


Phase 3: Not started. MLE and DOEPOD methods will be included (proposed work 
for FY18 & FY19). Sets precedent as the first complex calculation modules added 
to MAPTIS. FY16 presentation to national NDE council drew interest in 
application independent and standardized calculator. 


POD Studies 


Phase 1: Progress in Analyzing the OSMA NDE POD Data Set 
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Specimen Sets 


¢ About 860 metallic specimens were produced before 2011 with a selection of fatigue 
cracks, fastener hole cracks, lack of weld fusion, and electrical discharge machined (EDM) 
flaws at a cost of about $630,913. The metals used in this study are common throughout 
aerospace and include aluminum, titanium, nickel-chromium alloy, and stainless steel. 


¢ NDE was performed on these specimens to augment the NASA NDE Data Book, which 
feeds into Table 1, NASA-STD-5009. Specimens were examined with x-ray radiographic 
testing (RT) under differing film densities, digital radiography (DR), ultrasonic testing (UT) 
(including phased array UT), ET (including automated methods), fluorescent PT (L3 & L4), 
magnetic particle testing (MT), and visual testing (VT). Three of about 60 POD analysis 
ds been eaimleteuy 
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Specimen S ets — maintained by WSTF, available for PODs / recerts 


Aluminum Plates with Fatigue Cracks 
Box #1: AIA-1 to -16 

Box #2: AIA-19 to -46 

Box #3: AIA-47 to -74 

Box #4: AIA-75 to -102 

Titanium Plates with Fatigue Cracks 
Box #5: AIT-1 to -16 

Box #6: AIT-18 to -38 

Box #7: AIT-39 to -59 

Box #8: AIT-60 to -80 

Box #9: AIT-81 to -101 

Inconel Plates with Fatigue Cracks 
Box #10: All-1 to -16 

Box #11: All-18 to -33 

Box #12: All-34 to -50 

Box #13: All-51 to -68 

Box #14: All-69 to -85 

Box #15: All-86 to -101 


Aluminum Flat Plate w/ LOF Welds 

Box #16: BIA-1 to -6 —, 
Box #17: BIA-18L to 39L r 
Box #18: BIA-40L to 61L 
Titanium Flat Plate w/ LOF Welds 
Box #19: BIT-1 to 16 

Box #20: BIT-18 to 38(7) 

Box #21: BIT-40 to 61 

Inconel Welds w/ EDM Notches 
Box #22: BII-18 to 22, 24-25, & 27-37 
Welds w/ EDM Notches (SS, Al, Ti) 
Box #23: BIS-18 to 27, 29-30, & 32-37 
Box #24: BIA-18 to -37 

Box #25: BIT-18 to -37 


Inconel Tubes with Fatigue Cracks 


Box #26: Cll1-1 to 16 Peal 


Box #27: CIl1-17 to 33 
Stainless Tubes with Fatigue Cracks 
Box #28: CIS1-1 to 3, 9-16, & 18-20 
Box #29: CIS1-21 to 39 

Stainless Tubes with LOF Welds 
Box #30: CIS2-1 to 17 

Box #31: CIS2-18 to 33 


Inconel Tubes with Fatigue Cracks 
Box #32: Cl1-1L to 16L 


Box #33: Cl1-18L to 39L 

Box #34: Cl1-40L to 61L 

Inconel Tubes with LOF Welds 

Box #35: CIS1-18L to 33L 

Box #36: CIS1-40L to 61L 

Box #37: CIS1-62L to 83L 

Stainless Tubes with LOF Welds 
Box #38: CIS2-18L to 33L 

Box #39: CIS2-40L to 61L 

Box #40: CIS2-62L to 83L 

Cracked Fastener Holes, Aluminum 
Box #41: DIA-1 to 16 
Box #42: DIA-18 to 31 
Box #43: DIA-32 to 46 
Box #44: DIA-47 to 61 
Cracked Fastener Holes, Titanium 
Box #45: DIT-2 to 17 

Box #46: DIT-18 to 31 

Box #47: DIT-32 to 46 

Box #48: DIT-47 to 61 


Box1 and Box2 are radiographs 
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Inspection Data Produced 


90/95 POD 
Analyses : 
¥ Specimen Type 
101 Plates Ea. 61 Welds Ea. 20 Welds Ea. 33 Tubes 61 Tube Welds 61 Fastener 
i w/ EDM w/ LoF w/ EDM w/ EDM w/ LoF Hole Cracks 
Material - 
Al INCO Ti Al Al 
Mu 4 _ | 
Ti Ti 
——— nN—_ 7 


6 simi 90/95 POD — 90% flaws found w/ 95% confidence 
. | Assume this applies to all | EDM — Electrical Discharge Machining Notches 

2 material/sample LoF — Lack of Weld Fusion 

c ' an as ' Al — Aluminum 

9 ! combinations. ! x Kit = Inconel 

=) ' ' ;. : 

o i —_ Ti — Titanium 

a (Three Inspectors SS — Stainless Steel 

= | performed each method.) DT — Destructive Evaluation Data 


ET — Eddy Current Test Data 

PT — Fluorescent Dye Penetrant Test Data 
RT — Radiologic Test Data 

UT UT — Ultrasonic Test Data 
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MAPTIS 


MATERIALS AND PROCESSES TECHNICAL INFORMATION SYSTEM 


NDE POD Standards Library 


Phase 2: Creates a centralized repository for POD specimens/ NDE 
capabilities essentially providing a living, backed-up, and encrypted table 
for NASA-STD-5009 via MAPTIS. 


Also logs Phase 1 specimens for agency support and dissemination. 
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NDE Probability Standards Library 
NPSL) Database — Main Menu 


| MENU | search here.. searches all records and documents. Welcome to NDE Probability of Detection Standards Library (NPSL) 


Probability of Detection Calculator (In Work) J Capability Demonstration Specimens J Completed Capability Demonstration Studies a) NewRecord [i | fi home Je help | 


NDE Probability of Detection Standards Library 
OTey ile) (-iczve MOx-Ter-leyi inv PL-Vilelijie-laleli mei tlce (21) Purpose 


The library provides a living accounting of the capability demonstration studies and specimens used to 
establish nondestructive evaluation (NDE) method detection limits for fracture critical hardware across 
the agency. 


Probability of Detection Study How-To Guides Applicable Standards 


Per NASA-STD-5009: Special NDE consists of nondestructive inspections that must be capable of 
detecting crack-like flaws smaller than those detectable by Standard NDE (table 1 or table 2) or those 
that do not conform to the requirements for Standard NDE given in section 4.2, Standard NDE. All 
deviations from standard NDE shall be approved by the responsible NDE engineering organization, the 


Capability Demonstration Specimens RFCB, and Technical Authority 


Demonstration of Capability 
A 90/95 percent flaw detection capability shall be demonstrated before a Special NDE inspection can be 


implemented. The demonstration of the Special NDE inspection at a given crack size qualifies the 
Hh 7 Special NDE for implementation for the detection of cracks at the demonstrated size and larger. Once 
Probability of Detection Calculator (In Work) established and accepted, Special NDE qualification is generally limited to the equipment, procedure, 
and inspectors involved in the capability demonstration study unless approved by the responsible NDE 


engineering organization, the RFCB, and Technical Authority. 


Acknowledgements 


The library is maintained by Marshall Space Flight Center and managed by Johnson Space Center via 
White Sands Test Facility. Support for the library is provided by the NASA Office of Safety and Mission 
Assurance NDE Program. 


Points of Contact 


Project Manager: Charles Nichols, 575.524.5389, charles.nichols@nasa.gov 
Technical Lead: Rick Serwecki, 256.544.3977, rick.serwecki@nasa.gov 


Add Another User Helpful Points of Contact 


Under Construction. Likely to go live in June: 
https://maptis.ndc.nasa.gov/Athena/ViewRecords.aspx?pk=32 
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NPSL Database — How-To Guides  Daaty 


| MENU | search here.. searches all records and documents. Welcome to NDE Probability of Detection Standards Library (NPSL) 


Probability of Detection Calculator (In Work) Capability Demonstration Specimens Completed Capability Demonstration Studies a) New Record i | | n home y help | 


PROBABILITY OF DETECTION STUDY HOW-TO GUIDES 


starts with: | All| V] of Select All Generate Report 


Filter By ~ 


ASTM E 2862, Standard Practice for Probability of Detection Analysis for Hit/Miss Data 


Ed Generazio - DOEPOD NDE Capabilities Data Book 


EN-SB-08-012, USAF In-Service Inspection Flaw Assumptions for Metallic Structures 


Floyd Spencer - Nonparametric POD Estimation for Hit/Miss Data Model Comparisons 


Jennifer Brown - It takes more than a Statistician to do POD 


MIL-HDBK-1823A (2009), DoD Handbook for NDE System Reliability Assessments 


MIL-HDBK-5J, DoD Handbook for Metallic Materials and Elements for Aerospace Vehicle Structures 


NASA-STD-5009 Draft Appendix, Best Practices for POD Testing - Specimen Storage Best Practices 


NASA-STD-5009, Nondestructive Evaluation Requirements for Fracture-Critical Metallic Components 


Ward Rummell - Recommended Practices 


page 1 of 2(11 Records) <<first <prev a. 2 | next> | last >> | 
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NPSL Database — POD Specimens at 


New Record x 


Notify Me 


Record Name *: 


Specimens Flaws Delete Me Owner Location 
General Characteristics of Each Box of Specimens 


Representative Images * : Browse. 


Click "Browse..." and select one or more files (hold CTRL to select more than one) or drag 
and drop files here to upload 


Clear Files Specnnen rane... | yr oy 


Specimen type: iy] Specimen Outer Diameter: in |v] 
Material *: [set bay Specimen Inner Diameter: in [¥] 


Alloy*: | Select iv} Detailed Inventory: Byco. } 


Specimen Type *: | Select ina Click "Browse..." and select one or more files (hold CTRL to select more than one) or drag 


and drop files here to upload. 


Flaw Type * : { Select 
Specimen Width: in {[¥] 
Specimen Length: rs in [¥} 
Specimen Thickness: in [Vv 
= Clear Files 
Specimen Outer Diameter: in |¥] 
Date of Inventory * : a 
Specimen Inner Diameter: in [Vy] 


Inventory Conducted By*: | Select 


Detailed Inventory: Browse. 


Range of Specimen Serial Numbers [~ 


Delete, likely available in one month | Select v 
on 1/22: = 
Container Number * : 


* Indicates required attribute 


Click "Browse..." and select one or more files (hold CTRL to select more than one) or drag 
and drop files here to upload. 


CAPABILITY DEMONSTRATION SPECIMENS 
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NPSL Database — POD Studies (Draft) 


: New Record xX 


save’ 
Notify Me 


Record Name* : 


Capability... Certified ... 


NDE Capability Demonstration Study 


Location: [Seen SSCS 
NDEMethod: [Select == ss—<—s SL 
Mawisl*:(slet S 
Alloy*: (Stet SSF 


Files & Report * : Browse: 


Click "Browse..." and select one or more files (hold CTRL to select more than one) or drag 
and drop files here to upload. 


Clear Files] 


* Indicates required attribute 
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NPSL Database — Upload Templates 


NDE Probability of Detection Standards Library Record Import x 


| MENU | search here.. searches all records and documents. 


Templates Upload Dashboard 


Admin : 
vi lo : Capability Demonstration Speci Record Category:| Specimens [yy] 
Dashboard > . | 
Template Format: @Horizontal OVertical 
Import Import Templates Add User Additional Options: © @)Include all Attributes O Select Attributes 
My Settings Upload Calculator 


New Record Import Dashboard Guides 
POCs 


Specimens 
Study | 


Results in a MS Excel spreadsheet laid out 
to enable rapid, batched uploads. 


Users can just copy > paste into the 
appropriate column from existing tables. 
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NPSL Database — Search & Reporting Features 


maptis home | search | help | Tutorial 


inconel | a | Classic Search 


Material Record Count: 1 


Xinconel 
%Pneu Impact: Pressure 5000 to 10000 


Filter Results By 


lv] Metal Non-Metal ~ 
Mech Impact | where Pressure (psia) iy | 5000 to 10000 
Results 
= ae 1 Material matched your search 
Materials Definition ~ 
U U 
~~ ee ae nee Composition Specification 
Code (Designation) . P P P P 
Min Max 
10129. INCONEL™ 718 -423 1120 50-55NI 17-21CR 17FE .2-.8AL .006B .08C 1CO .3CU .35MN MBO0170-075 


BAR ST 1750F 


Generate PDF Generate Report 


Example: Material Selection Database, Link: https://maptis.ndc.nasa.gov/matsel/ 


2.8-3.3MO 4.75-5.5NB .015P .015S .35S1 .65-1.15T1 
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NPSL Database — Access Controls 


NDE Probability of Detection Standards Library Project Admin 


Project Data Map User Rights Access Control 


Enter Name, User ID or Email Address: | eric burke 


LAST NAME FIRST NAME E-MAIL 


Burke Eric eric.r.burke@nasa.gov Select 


User ID: erburke 
Name: Eric R Burke 
Email: eric.r.burke@nasa.gov 
User Rights: [[] Project Admin 
[V7] Edit Records 
[¥] Delete Records 
(_] Modify Access Controls 
[V7] View All Records 
(J User Admin 
[| Import Module 
{_] Bulk Delete 


{_] Dashboard Module 


Update 
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Schedules/Muilestones 


¥Y FY15: Inventory all data; identify and fill gaps, should they exist. Provide a Best 
Practices appendix to NASA-STD-5009. Include recommendations for long-term 
sample storage. 


Vv FY16-17: Analyze and log the following NNWG POD standards —NDE Methods: 
UT, RT, ET, & PT. Not practical to accomplish in one year under given resources. 


¢ About 60 studies would need to be performed. Three studies were wrapped up. 
¢« ~101 flat plates with fatigue cracks (Al, INCO, & Ti) 
¢ 61 plate weld specimens with lack of fusion (Al & Ti) 
¢ 20 plates welded with an EDM notch specimens (Al, INCO, SS & Ti) 
¢« ~33 1” or 2” tubes welded with an EDM notch (INCO, SS) 
* ~611” or 2” tubes welded with lack of fusion (INCO & SS) and 
¢ 61 fastener hole crack specimens (Al & Ti) 


LIFY17+: Alpha test/approve NPSL entry and query capabilities. Create a 
platform-independent POD ana lysis tool Extract, interpret, and integrate macros from DOEPOD into NPSL. 


* Code in MLE as presented in NASA/TM—2014-218183, April 2014. 
e Add internal and external validation test for MLE estimates (these have been draft coded into DOEPOD, but not released). 


LI Proposed for FY18-19: Test and approve NPSL entry and query capabilities. 
Create a platform-independent POD analysis tool 


* Codein MLE as presented in NASA/TM—2014-218183, April 2014. 
¢ — Add internal and external validation test for MLE estimates (these have been draft coded into DOEPOD, but not released). 
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Customers 


NASA Infusion 


1. Having a central catalog of POD studies provides a number of benefits: tracking POD study values across a 
wide variety of parameters to establish precedence via a living database, auditable sources for POD 
studies and inspector certs, alerting Center reps to looming inspector recertification expiration (2-year 
certs per NASA-STD-5009), the ability to learn how other Centers qualify NDE, etc. 


2. Once Center surveys are completed, the database will serve as a centralized NASA library for locating the 
specimens and resources needed to perform capability demonstration studies. As these specimens can be 
very expensive to produce, collaboratively sharing resources and reducing duplicated efforts has the 
potential for saving programs and agency NDE practitioners thousands of dollars each year. 


3. Later, addition of a POD calculator to the database will reduce the practice of using multiple tools across 
NASA to run estimations. 


U.S. Air Force 
¢ Though the necessary validation, security protocols, and database population will take some time, the 
project team wishes to make NPSL accessible by agency partners across the country. This is unprecedented. 


There is some USAF interest in an updated database based on Berens and Hovey POD v3 implementing MIL- 
HDBK-1823, but this has not yet come to pass. 


U.S. Navy 


¢ NASA collaboration through NAVSEA/NUWC is a possibility on some submarine work, however the USAF 
remains the most prolific user of POD throughout the U.S. military. 
AT = 
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Products 


Publications 


di 


Binomial Test Method for Determining Probability of Detection Capability for Fracture 


Critical Applications (http://ntrs.nasa.gov/search.jsp?R=20110015149) 


2. Directed Design of Experiments for Validating Probability of Detection Capability of 
NDE Systems (DOEPOD), NASA TM 2015-218696 
(http://ntrs nasaleeu/are haeieeey HED ntrsinasa @OW7701S001 398 Neri 

3. — Interrelationships Between Probability of Detection Methodologies, NASA/TM-—2014- 
218183 init pf Mia, Oy ice Vie asa/easi-nitrearieeemalmn.A0005 Sac) 

4. NASA DOEPOD NDE Capabilities Data Book, NASA TM 2015-218770 
(http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20150013943.pdf) 

5. | OSMA’s NDE Program Published Capabilities Data Book, OSMA Website 

Patents 

1. US Patent 8108178, Directed Design of Experiments for Validating Probability of 


Detection Capability of a Testing System 
(http://www.anypatents.com/patents/US8108178) 
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Metrics 


ee | Technology Maturity == 


| Current: 4. 4| Estimated End: 9 


aoe aa cae 
Research ment cae 


Technology Areas 


Materials, Structures, Mechanical 

Systems and Manufacturing (TA 

12) 

— Nondestructive Evaluation 
Sensor and Method (TA 
12.4.5.1) 


Cross-Cutting (no content) (TA 
12.5) 
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U.S. WORK LOCATIONS AND KEY PARTNERS 


BUS. States With Work vf Lead Center: 
White Sands Test Facility 


’ Supporting Centers: 

* Goddard Space Flight Center 
* Johnson Space Center 

# Langley Research Center 
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O&A 


¢ For further information, refer to the following report: 


Link: https://techport.nasa.gov/view/32921 


¢ Point of Contact: 


Charles Nichols, PM 
NASA JSC White Sands Test Facility 
Telephone: (575) 524-5389 


charles.nichols @nasa.gov 


